Non-transitory computer-readable storage medium storing program, terminal device and method

ABSTRACT

A non-transitory computer-readable storage medium stores instructions including first generating processing, first determination processing, second determination processing, second generating processing, and third determination processing for a terminal device with a camera. The first generating processing generates extracted image data from captured image data acquired by the camera. The first determination processing determines representative luminance values respectively representing luminances of the extracted image data. The second determination processing determines luminance correction values with respect to each of the plurality of extracted image data, based on a reference luminance value and the representative luminance values. The second generating processing generates composite image data from the extracted image data. The luminance of each extracted image data has been corrected using the luminance correction values. The third determination processing determines, based on the luminance correction values, an exposure correction value to correct an exposure of the camera.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to Japanese Patent Application No.2014-072011 filed Mar. 31, 2014, the content of which is herebyincorporated herein by reference.

BACKGROUND

The present disclosure relates to a non-transitory computer-readablestorage medium storing a program, a terminal device and a method thatare capable of performing a remote conference by performing thetransmission and reception of various data via a network.

In known art, a terminal device is known that can be connected to aplurality of terminal devices, such as a mobile terminal, a personalcomputer (PC) and the like, via a network, and can perform a remoteconference by the transmission and reception of various data, includingimage data, between the terminal devices. For example, as the terminaldevice, a video communication device is known that transmits a videosignal of a captured image captured by an imaging device (a camera) toanother video communication device. When the video communication devicereceives, from the other video communication device, a control signalthat specifies an area of part of the captured image, the videocommunication device extracts an image of the specified area from thecaptured image. The video communication device transmits a video signalof the extracted image to the video communication device thattransmitted the control signal specifying the area. Further, when thevideo communication device receives, from the other video communicationdevice, a control signal that specifies a plurality of areas in thecaptured image, the video communication device respectively extractsimages of the specified plurality of areas from the captured image. Thevideo communication device generates a composite image in which theextracted plurality of images are combined as a single image. The videocommunication device transmits a video signal of the composite image tothe video communication device that transmitted the control signalspecifying the areas.

SUMMARY

Aspects described herein may provide a non-transitory computer-readablestorage medium storing computer-readable instructions. When executed bya processor of a terminal device, which is configured to performtransmission and reception of various data with a plurality of terminaldevices connected via a network, perform processes. The processesinclude first generating processing, first determination processing,second determination processing, storage processing, second generatingprocessing, and third determination processing. The first generatingprocessing generates extracted image data from captured image dataacquired by a camera connected to the terminal device. The extractedimage data correspond to portions in the captured image data. Theextracted image data includes a target to be extracted from the imagedata. The first determination processing determines a plurality ofrepresentative luminance values respectively representing luminances ofthe extracted image data. The second determination processing determinesa plurality of luminance correction values with respect to each of theplurality of extracted image data, based on a reference luminance valueand the plurality of representative luminance values determined in thefirst determination processing. The reference luminance value is areference value of luminance. The storage processing stores theplurality of luminance correction values determined in the seconddetermination processing in a storage device provided in the terminaldevice. The second generating processing generates composite image datafrom the extracted image data. The luminance of each of the plurality ofextracted image data has been corrected using the plurality of luminancecorrection values stored in the storage device in the storage processingwith respect to each of the extracted image data to which the luminancecorrection values respectively correspond. The third determinationprocessing determining, based on the plurality of luminance correctionvalues stored in the storage device in the storage processing, anexposure correction value to correct an exposure of the camera.

Aspects described herein may also provide a terminal device performingtransmission and reception of various data with a plurality of terminaldevices configured to be connected via a network. The terminal deviceincludes a processor and a memory. The memory stores computer-readableinstructions. The instructions, when executed by the processor, performprocesses. The processes include first generating processing, firstdetermination processing, second determination processing, storageprocessing, second generating processing, and third determinationprocessing. The first generating processing generates extracted imagedata from captured image data acquired by a camera connected to theterminal device. The extracted image data correspond to portions in thecaptured image data. The extracted image data includes a target to beextracted from the image data. The first determination processingdetermines a plurality of representative luminance values respectivelyrepresenting luminances of the extracted image data. The seconddetermination processing determines a plurality of luminance correctionvalues with respect to each of the plurality of extracted image data,based on a reference luminance value and the plurality of representativeluminance values determined in the first determination processing. Thereference luminance value is a reference value of luminance. The storageprocessing stores the plurality of luminance correction valuesdetermined in the second determination processing in the memory. Thesecond generating processing generates composite image data from theextracted image data. The luminance of each of the plurality ofextracted image data has been corrected using the plurality of luminancecorrection values stored in the memory in the storage processing withrespect to each of the extracted image data to which the luminancecorrection values respectively correspond. The third determinationprocessing determines, based on the plurality of luminance correctionvalues stored in the memory in the storage processing, an exposurecorrection value to correct an exposure of the camera.

Aspects described herein may further provide a method executed by aprocessor of a terminal device, which performs transmission andreception of various data with a plurality of terminal devicesconfigured to be connected via a network. The method includes firstgenerating extracted image data from captured image data acquired by acamera connected to the terminal device. The extracted image datacorrespond to portions in the captured image data. The extracted imagedata includes a target to be extracted from the image data. The methodalso includes first determining a plurality of representative luminancevalues respectively representing luminances of the extracted image data.The method also includes second determining a plurality of luminancecorrection values with respect to each of the plurality of extractedimage data, based on a reference luminance value and the plurality ofrepresentative luminance values determined in the first determining. Thereference luminance value is a reference value of luminance. The methodalso includes storing the plurality of luminance correction valuesdetermined in the second determining in a storage device provided in theterminal device. The method also includes second generating compositeimage data from the extracted image data. The luminance of each of theplurality of extracted image data has been corrected using the pluralityof luminance correction values stored in the storage device with respectto each of the extracted image data to which the luminance correctionvalues respectively correspond. The method also includes thirddetermining, based on the plurality of luminance correction valuesstored in the storage processing, an exposure correction value tocorrect an exposure of the camera.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will be described below in detail with reference to theaccompanying drawings in which:

FIG. 1 is a schematic overview of a remote conference system 100;

FIG. 2 is a flowchart of a remote conference program that is executed bya terminal 3;

FIG. 3 is a flowchart of video transmission/reception processing in theexecution of the remote conference program;

FIG. 4 is flowchart of camera control processing in the execution of theremote conference program;

FIG. 5 is a flowchart of reference luminance value determinationprocessing in the execution of the remote conference program;

FIG. 6 is a flowchart of discrete image luminance adjustment processingin the execution of the remote conference program;

FIG. 7 is a flowchart of image capture exposure adjustment processing inthe execution of the remote conference program;

FIG. 8 is a diagram showing an example of an image 50 captured by acamera 44 in a remote conference;

FIG. 9 is a diagram showing an example of an image represented byextracted image data 66 to 69; and

FIG. 10 is a diagram showing an example of a composite image 70.

DETAILED DESCRIPTION

Within an angle of view of a camera, an image capture target of thecamera is influenced by light from a light source that illuminates theangle of view of the camera, depending on various factors, such as aposition, orientation, size etc. of the image capture target of thecamera. When there are a plurality of image capture targets, theinfluence of the light on the individual image capture targets maydiffer for each of the image capture targets. For example, when theimage capture targets are human faces, as the influence of the light oneach of the individual faces is different as a result of variousfactors, a single captured image may include images of light faces andimages of dark faces. The light source also causes the light toinfluence other aspects, such as the background and stationary objectsetc., and not just the image capture targets. Further, a direction ofthe light from the light source is the same with respect to everythingwithin an angle of view of the camera. As a result, even when thecaptured image includes the images of light faces and the images of darkfaces, changes in luminance within the image form a natural image inwhich, in the image as a whole, the luminance changes continuously basedon the direction of the light.

However, when a plurality of areas that respectively include individualimage capture targets are extracted from the captured image, dependingon the influence of the light on the image capture targets included inthe extracted areas, variations in brightness occur in the individualimages. As a result, when the extracted images are simply combined as asingle image, in the resulting composite image, the luminance varies foreach of areas occupied by the original extracted images. Therefore,changes in the luminance within the combined image are not based on thedirection of the light in the image as a whole, and it is possible thatan unnatural image is obtained in which the changes in luminance are notcontinuous.

In addition, when the images of the light faces and the images of thedark faces are included in the single composite image, as it isdifficult to distinguish the expressions of the faces, a problem mayarise in which it is difficult to use the composite image in a remoteconference. If a terminal device performs luminance correction withrespect to each of the images extracted from the captured image, it ispossible to generate a composite image in which variations in theluminance are reduced. However, the captured image that is used in theremote conference is a frame image that is continuously displayed asvideo. Thus, if luminance correction is performed on each frame, it ispossible that a CPU processing load may be increased in the terminaldevice.

The present disclosure, for example, provides a non-transitorycomputer-readable storage medium storing a program, a terminal deviceand a method that are capable of correcting the luminance of imagesextracted from a captured image and generating a composite image, andare also capable of correcting a camera exposure using an adjustmentvalue that accords with a luminance correction value.

Hereinafter, an embodiment of the present disclosure will be explainedwith reference to the drawings. The drawings referred to are used toexplain technological features that can be adopted by the presentdisclosure. Configurations of devices noted in the drawings, andflowcharts of various processing etc. are not limited only to theexamples given and are simply explanatory examples.

A remote conference system 100 that includes terminals 3 according tothe present embodiment will be explained with reference to FIG. 1. Theremote conference system 100 is a system that can hold a remoteconference (e.g., web conference or a television conference) that usesimages and audio. The remote conference system 100 includes a conferenceserver 1 and the plurality of terminals 3. The conference server 1 andthe plurality of terminals 3 are connected each other via a network 8.In the remote conference system 100, the plurality of terminals 3respectively participate, virtually and via the network 8, in aconference of a conference room that is virtually established by theconference server 1. Each of the terminals 3 is used by a userparticipating in the remote conference. The conference server 1 controlsthe remote conference that is being held between the plurality ofterminals 3.

The remote conference system 100 shown in FIG. 1 is a system in which,as an example, the single conference server 1 and three of the terminals3 are respectively connected via the network 8. The conference server 1virtually establishes a conference room, as a forum for the remoteconference being held using the plurality of terminals 3, and manageseach of the terminals 3 entering or leaving the conference room. Thevirtual conference room is identified by a predetermined conference roomID. The plurality of terminals 3 participate in the same remoteconference that is identified by the predetermined conference room ID.Further, the plurality of terminals 3 are respectively identified byuser IDs. For example, the user IDs of the plurality of terminals 3participating in the same remote conference are associated with thepredetermined conference room ID and stored in a storage device (such asa RAM, or a hard disk drive (hereinafter referred to as an “HDD”), forexample) of the conference server 1. The plurality of terminals 3 areconnected to the conference server 1 via the network 8 and transmit dataof images (video) captured by the respective terminals 3 and audiocollected by the respective terminals 3 to the conference server 1,along with the conference room ID. The conference server 1 transmitsdata of the received images (video) and audio to each of the pluralityof terminals 3 that are identified by the user IDs associated with theconference room ID that is received along with the data of the imagesand audio. With this configuration, the plurality of terminals 3 canmutually transmit and receive data of images (video) and audio acquiredby each of the terminals 3 and can perform the remote conference throughreal time conversation. The conference server 1 is configured in asimilar manner to a known work station or personal computer (PC)configuration, except for the point that various processing programs,which allow a remote conference between the plurality of terminals 3that are connected to the conference server 1 to be performed, arestored in the storage device of the conference server 1. An explanationof the electrical configuration of the conference server 1 is thereforeomitted here.

A monitor 41, a microphone 43, a camera 44 and a speaker 45 etc. areconnected to the terminal 3. A PC can be used as the terminal 3, forexample. A portable terminal, such as a smart phone, a tablet terminalor the like may also be used as the terminal 3. At the time of theremote conference, one of the terminals 3 transmits image data capturedby the camera 44 and audio data collected by the microphone 43 to theconference server 1, along with the conference room ID. The image dataand the audio data are transmitted via the conference server 1 to theother terminals 3 that are participating in the remote conferenceidentified by the conference room ID. Further, the one terminal 3displays an image on the monitor 41 based on image data received fromthe other terminals 3 via the conference server 1. The one terminal 3outputs audio from the speaker 45 based on audio data received from theother terminals 3. As a result, in the remote conference system 100, theimage data and audio data transmitted from the plurality of terminals 3are shared between the plurality of terminals 3. Even when all theparticipants (all the users of the terminals 3) in the remote conferencesystem 100 are not in the same place, the remote conference system 100can provide a forum for a smooth conference between the participants.Note that, as the control of the remote conference performed by theconference server 1 in the remote conference system 100 is the same asin known art, a detailed explanation thereof is omitted here. Further,in the present embodiment, of the image data and the audio datatransmitted and received between the terminals 3, the control of thetransmission and reception of the image data will be explained, and anexplanation of the control of the transmission and reception of theaudio data will be simplified.

An electrical configuration of the terminal 3 will be explained. Theterminal 3 is a PC having a known configuration, and is provided with aCPU 31 that controls the terminal 3. The CPU 31 is connected to a chipset 35, and is electrically connected to a ROM 32, a RAM 33 and adisplay control portion 34, via the chip set 35. The chip set 35 isconnected to a chip set 36. The CPU 31 is electrically connected to anHDD 37, a communication I/F 38 and a USB I/F 39 via the chip set 36.

The chip set 35 is a series of circuits that manage the transmission andreception of data between the CPU 31 and the ROM 32, the RAM 33 and thedisplay control portion 34. A boot program, a BIOS and the like arestored in the ROM 32. Various temporary data are stored in the RAM 33.The display control portion 34 controls display of images on the monitor41. The chip set 36 is a series of circuits that manage the transmissionand reception of data between the CPU 31 and the HDD 37, thecommunication I/F 38 and the USB I/F 39. An OS, software that causes theterminal 3 to function as a client (a remote conference program thatwill be explained later) and other various applications and data etc.are stored in the HDD 37. The communication I/F 38 is connectedwirelessly or by a hard-wired connection to the network 8 and is aninterface to perform data communication.

It should be noted that the remote conference program that will beexplained later is compiled as a code depending on the terminal 3, andis saved in a computer-readable storage device that is provided in afile server (not shown in the drawings), the conference server 1 or thelike. The remote conference program is provided by being transmittedfrom the file server, the conference server 1 etc. to each of theterminals 3 as a transitory transmission signal, via atelecommunications line of the network 8 or the like. Alternatively, theremote conference program may be provided by being stored in anon-transitory computer-readable storage medium (not shown in thedrawings), such as a CD-ROM, a DVD-ROM, a flash ROM or the like, and bythe CPU 31 of the terminal 3 using a reader (not shown in the drawings)that can be connected to the terminal 3 to read out the remoteconference program from the non-transitory computer-readable storagemedium. The CPU 31 stores the remote conference program in a storagedevice that can be read by a computer provided in the terminal 3. In thepresent embodiment, the storage device is the HDD 37, but the storagedevice is not limited to the HDD 37 and may be a non-volatile storagedevice, for example, such as a solid state drive (SSD) or a flash ROM orthe like. In other words, it is preferable for a non-transitorycomputer-readable storage medium, which can hold data irrespective of alength of time of storing the data, to be used as the storage device.

The USB I/F 39 is an interface to perform communication based on USBstandards. The CPU 31 is electrically connected to an input portion 42,the microphone 43, the camera 44 and the speaker 45 via the USB I/F 39.The input portion 42 is a device, such as a keyboard, a mouse etc., thatis used to input an operation on the terminal 3. The microphone 43collects audio around the terminal 3, and transmits A/D converted(sampled) audio data to the OS via the USB I/F 39. The camera 44captures images of the scenery around the terminal 3 and transmits A/Dconverted image data to the OS via the USB I/F 39. The camera 44includes a two-dimensional imaging element, such as a CCD or a CMOS, anda lens system that focuses external light on the imaging element to forman image. The camera 44 further includes a conversion circuit thatamplifies analog data output from the imaging element, performs A/Dconversion and generates image data that is digital data. The speaker 45performs D/A conversion on audio data received from the OS via the USBI/F 39 and outputs audio.

In the remote conference system 100, the CPU 31 of the terminal 3performs the following various processing in accordance with the remoteconference program. Using a known encoder, the CPU 31 encodes the audiodata collected by the microphone 43 in accordance with predeterminedaudio parameters, and generates encoded audio data. The predeterminedaudio parameters are encoding conditions and are, for example, “useHE-AAC v2 as codec, set bit rate to 48 kbps (CBR) and set sampling rateto 24 kHz” or the like. The CPU 31 transmits the generated encoded audiodata to the other terminals 3 via the conference server 1. When the CPU31 of each of the other terminals 3 receives the encoded audio data, theCPU 31 uses a known decoder to decode the encoded audio data andgenerates audio data, then reproduces the audio data from the speaker45.

In a similar manner, the CPU 31 uses a known encoder to compress theimage data captured by the camera 44 in accordance with predeterminedvideo parameters and generates encoded image data. The predeterminedvideo parameters are encoding conditions and are, for example, “useH.265 as codec, set resolution to 1280×720p, 30 fps, and set bit rate to1 Mbps (VBR)” or the like. The CPU 31 transmits the generated encodedimage data to the other terminals 3 via the conference server 1. Whenthe CPU 31 of each of the other terminals 3 receives the encoded imagedata, the CPU 31 uses a known decoder to decode the encoded image dataand generates image data, then displays an image on the monitor 41.

Each of the above-described encoders and decoders is a program modulethat the CPU 31 calls up as appropriate in order to perform processingin the execution of the remote conference program. The encoders and thedecoders are included in the remote conference program and are installedin the terminal 3 in advance. The basic software of the PC is amulti-tasking OS, and the remote conference program, the encoder and thedecoder etc. can be executed by parallel processing. Note that, when thebasic software is not a multi-tasking OS, each of the program modulesmay be activated based on an event interrupt from hardware or the like.

Further, various drivers (software) that respectively control theoperations of various devices, such as the speaker 45, the camera 44,the microphone 43 and the display control portion 34, are each builtinto the basic software of the PC. Each of the drivers is installed inadvance in the HDD 37, and is read into the RAM 33 at the time of the OSstart up. The drivers are constantly executed by parallel processing.Via the OS, the CPU 31 respectively transmits a command to drive each ofthe drivers to the drivers corresponding to each of the devices. Each ofthe drivers outputs a control instruction to the respectivelycorresponding device in accordance with the command, and therebycontrols the operation of each of the devices.

Among the various drivers, a camera driver that controls the operationof the camera 44 outputs a control instruction to the camera 44 inaccordance with a control command transmitted by the CPU 31 via the OS,and thereby controls the operation of the camera 44. The controlinstruction output to the camera 44 by the camera driver includes anexposure setting instruction that sets an exposure of video captured bythe camera 44. The exposure refers to a total amount of light to whichthe imaging element (not shown in the drawings), such as the CCD,provided in the camera 44 is exposed when the image data is captured.Therefore, the exposure also corresponds to the brightness of the video(image) captured by the imaging element. The exposure is determined by acombination of a lens aperture (f-number), a light exposure time andsensitivity of the imaging element etc.

In the execution of the remote conference program that will be explainedlater, the CPU 31 transmits, to the camera driver, a change command tochange the exposure of the camera 44. The change command is a commandthat includes an exposure correction value indicating a change amount ofthe luminance value of the image data acquired from the camera 44 by theCPU 31. Note that luminance values of respective pixels that configurethe image data are referred to collectively as the image data luminancevalue. The camera driver transmits exposure parameters (the f-number,the light exposure time and the sensitivity of the imaging elementetc.), which are used to set the exposure of the camera 44, to thecamera 44 as the exposure setting instruction. When the camera driverreceives the change command from the CPU 31, the camera driver changesthe exposure parameters in accordance with the exposure correction valueand outputs the exposure parameters after the change to the camera 44 asthe exposure setting instruction. The camera 44 changes the exposure inaccordance with the exposure setting instruction and outputs thecaptured image data. It should be noted here that the camera driver neednot correct the exposure of the camera 44 only through hardwarecorrection due to the changes in the exposure parameters, but may alsoperform software correction by executing processing to change theluminance value on the image data captured by the camera 44.

Hereinafter, processing that is performed by the CPU 31 of the terminal3 in the execution of the remote conference program and that is shown inFIG. 2 to FIG. 7 will be explained while referring as necessary to FIG.8 to FIG. 10. Based on an operation input on the input portion 42, theCPU 31 of the terminal 3 reads out main processing (refer to FIG. 2) ofthe remote conference program from the HDD 37 and executes the mainprocessing. Using initial settings at the time of execution of theremote conference program, the CPU 31 secures storage areas of atransmission buffer and a reception buffer respectively in the RAM 33.The transmission buffer is a storage area which temporarily stores theencoded audio data and the encoded image data transmitted by the CPU 31to the other terminals 3 via the conference server 1. The receptionbuffer is a storage area which temporarily stores the encoded audio dataand the encoded image data received by the CPU 31 from the otherterminals 3 via the conference server 1.

Note that, in the present embodiment, the CPU 31 of the one terminal 3stores the encoded audio data and the encoded image data generated inaccordance with the execution of the remote conference program in thetransmission buffer. In accordance with processing of the OS, the CPU 31transmits the encoded audio data and the encoded image data stored inthe transmission buffer to the conference server 1. The conferenceserver 1 transmits the encoded audio data and the encoded image datareceived from the one terminal 3 to the other terminals 3. In a similarmanner, the conference server 1 transmits the encoded audio data and theencoded image data received from the other terminals 3 to the oneterminal 3. In accordance with processing of the OS, the CPU 31 of theone terminal 3 stores the encoded audio data and the encoded image datareceived from the conference server 1 in the reception buffer. The CPU31 reads out the encoded audio data from the reception buffer inaccordance with the execution of the remote conference program, andoutputs audio to the speaker 45 based on the audio data. In a similarmanner, the CPU 31 reads out the encoded image data from the receptionbuffer and causes video to be displayed on the monitor 41 based on theimage data. Note that the CPU 31 of the one terminal 3 may transmit theencoded audio data and the encoded image data directly to each of theterminals 3, including to the conference server 1.

As shown in FIG. 2, based on the operation input on the input portion42, the CPU 31 of the one terminal 3 determines whether a command toconnect the one terminal 3 to the conference server 1 has been received(step S1). More specifically, the CPU 31 receives, via the input portion42, an input of the ID of the virtual conference room into which theuser wishes to enter. For example, before the planned start of theremote conference, an electronic mail is transmitted from the conferenceserver 1 addressed to each of electronic mail addresses of each user ofthe terminals 3 that are planning to participate in the remoteconference. The electronic mails include a uniform resource locator(URL) for the remote conference in which the terminals 3 are toparticipate. The URL includes, for example, the virtual conference roomID as a query parameter. When an operation to select the URL is receivedby the input portion 42, the CPU 31 acquires the conference room IDinput by the user. Based on the operation on the input portion 42, theCPU 31 stands by until the conference room ID is acquired (no at stepS1).

When, based on the operation on the input portion 42, the CPU 31acquires the conference room ID (yes at step S1), the CPU 31 attempts toconnect to the conference server 1 via the network 8 (step S2). When theconnection with the conference server 1 is established, the CPU 31transmits an entry request to the conference server 1. The entry requestincludes the conference room ID and the user ID. When the conferenceroom ID included in the entry request matches the predeterminedconference room ID stored in the storage device of the conference server1, the user ID included in the entry request is associated with thepredetermined conference room ID and the associated data is stored in astorage device (not shown in the drawings) of the conference server 1.Then, based on the entry request, a notification is generated in theconference server 1 that allows participation in the conference in theexisting conference room identified by the predetermined conference roomID. Further, when the conference room ID included in the entry requestis not stored in the storage device of the conference server 1, in theconference server 1, the user ID included in the entry request isassociated with the conference room ID and the associated data is storedin the storage device of the conference server 1. Then, based on thestored conference room ID, a virtual conference room is generated in theconference server 1, and, based on the entry request, a notification isgenerated that allows participation in the conference in this conferenceroom. The CPU 31 receives the notification from the conference server 1,as a response to the entry request and that allows participation in theconference. The CPU 31 participates in the remote conference and allowsthe user to enter into the conference room that has been virtuallygenerated by the conference server 1.

The CPU 31 starts camera control processing (step S3). The cameracontrol processing is processing in which the CPU 31 causes the cameradriver to activate the camera 44 and in which composite image data (tobe explained later) is generated based on image data captured by thecamera 44. The camera control processing is continuously executed inparallel with the main processing of the remote conference program whilethe terminal 3 is participating in the remote conference. Details of thecamera control processing will be explained later.

The CPU 31 starts video transmission/reception processing (step S5). Thevideo transmission/reception processing is processing in which theencoded image data are transmitted and received between the one terminal3 and the other terminals 3 via the conference server 1. The videotransmission/reception processing is continuously executed in parallelwith the main processing of the remote conference program while theterminal 3 is participating in the remote conference. Details of thevideo transmission/reception processing will be explained later.

The CPU 31 starts audio transmission/reception processing (step S6). Theaudio transmission/reception processing is processing in which theencoded audio data are transmitted and received between the one terminal3 and the other terminals 3 via the conference server 1. The audiotransmission/reception processing is continuously executed in parallelwith the main processing of the remote conference program while theterminal 3 is participating in the remote conference.

As described above, in accordance with the OS, the CPU 31 receives theencoded audio data transmitted by the other terminals 3 via theconference server 1, and stores the received encoded audio data in thereception buffer. In the audio transmission/reception processing, theCPU 31 inputs the encoded audio data that has been read out from thereception buffer into the audio decoder, and acquires the audio datadecoded according to the predetermined audio parameters. The CPU 31stores the audio data in a predetermined storage area of the RAM 33. TheCPU 31 transmits a command to the driver of the speaker 45 via the OS,and causes the audio based on the audio data to be output from thespeaker 45. Further, the CPU 31 transmits a command to the driver of themicrophone 43 via the OS, and causes the audio data of the audiocollected by the microphone 43 to be stored in a predetermined storagearea of the RAM 33. The CPU 31 inputs the audio data into the audioencoder, and acquires the encoded audio data that has been encodedaccording to the predetermined audio parameters. The CPU 31 stores theencoded audio data in the transmission buffer. In accordance with theOS, the CPU 31 transmits the encoded audio data stored in thetransmission buffer to the other terminals 3 via the conference server1. The CPU 31 continuously performs each of the above-describedprocessing during the participation in the remote conference.

Next, the CPU 31 determines whether a command has been input toterminate the remote conference (step S7). When the command to terminatethe remote conference has not been input (no at step S7), the processingstands by. During stand-by, the CPU 31 repeatedly and continuouslyperforms the camera control processing, the video transmission/receptionprocessing and the audio transmission/reception processing.

When the input portion 42 in one of the other terminals 3 has receivedthe input of the operation to terminate the remote conference, forexample, the conference server 1 transmits a notification that theremote conference is to be terminated to the one terminal 3. When theCPU 31 receives from the conference server 1 the notification that theremote conference is to be terminated, the CPU 31 determines that therehas been input of a command to terminate the remote conference (yes atstep S7). The CPU 31 also determines that there has been input of thecommand to terminate the remote conference when the input portion 42 inthe one terminal 3 has received the input of the operation to terminatethe remote conference.

The CPU 31 executes conference termination processing (step S8). The CPU31 transmits a command to the driver that controls the operation of themicrophone 43 to stop the processing that stores the audio data in thepredetermined storage area of the RAM 33. The CPU 31 transmits a commandto the driver that controls the operation of the speaker 45 to stop theprocessing that outputs the audio from the speaker based on the audiodata. The CPU 31 transmits a command to the camera driver to stop thedriving of the camera 44. The CPU 31 transmits a command to the driverthat controls the operation of the display control portion 34 to stopthe processing that displays the image on the monitor 41 based on theimage data. The CPU 31 generates a notification event that notifies thetermination of the remote conference. The CPU 31 transmits an exitrequest to the conference server 1, via the network 8, requesting exitfrom the virtual conference room. The exit request includes theconference room ID and the user ID. In the conference server 1, based onthe conference room ID and the user ID included in the exit request,processing is performed that causes the terminal 3 identified by theuser ID to exit the virtual conference room. In the conference server 1,the user ID associated with the predetermined conference room ID storedin the storage device is deleted. Then, in the conference server 1, anotification is generated based on the exit request, which notifies thatthe exit from the conference room is complete. Note that, in theconference server 1, when all of the user IDs associated with thepredetermined conference room ID have been deleted, the virtualconference room is removed and the predetermined conference room ID isdeleted from the storage device of the conference server 1. The CPU 31receives the notification notifying that the exit from the conferenceroom is complete from the conference server 1, as a response to the exitrequest. The CPU 31 ends the connection to the conference server 1 viathe network 8 and terminates the execution of the remote conferenceprogram.

Next, the video transmission/reception processing that is started by theCPU 31 at step S5 of the main processing of the remote conferenceprogram will be explained in detail. As shown in FIG. 3, in the videotransmission/reception processing, the CPU 31 determines whether encodedimage data is stored in the reception buffer (step S11). When, theencoded image data received by the OS from the conference server 1 isnot stored in the reception buffer (no at step S11), the CPU 31 advancesthe processing to step S13. When, the encoded image data is stored inthe reception buffer (yes at step S11), the CPU 31 reads out the encodedimage data from the reception buffer. The CPU 31 inputs the encodedimage data into the image decoder, and acquires the image data decodedin accordance with the predetermined video parameters. The CPU 31transmits the image data to the display control portion 34 and causesthe image data to be displayed on the monitor 41 (step S12). Video basedon the image data captured by the other terminals 3 is displayed on themonitor 41. Further, in the camera control processing that will beexplained later (refer to FIG. 4), the CPU 31 also transmits the imagedata captured by the camera 44 of the one terminal 3 to the displaycontrol portion 34, and causes the captured image data to be displayedon the monitor 41 alongside the image data of the other terminals 3.Note that, in the camera control processing to be explained later (referto FIG. 4), the CPU 31 generates composite image data based on the imagedata, in accordance with conditions. When the composite image data isgenerated, the CPU 31 transmits the composite image data to the displaycontrol portion 34 in place of the image data captured by the camera 44of the one terminal 3, and causes the composite image data to bedisplayed on the monitor 41 alongside the image data of the otherterminals 3.

It should be noted that the CPU 31 may display each of the image datareceived from the plurality of terminals 3 via the conference server 1alongside each other on the monitor 41 using individual window displays.Alternatively, the CPU 31 may align the plurality of image data as atile formation and combine them as single image data and display them onthe monitor 41. Alternatively, a CPU (not shown in the drawings) of theconference server 1 may perform processing to align the respective imagedata of the plurality of terminals 3 as a tile formation and combinethem as single image data, encodes the single image data as the encodedimage data and transmit the encoded image data to the one terminal 3.

The CPU 31 advances the processing to step S13 and determines whether itis a timing to transmit the image data captured by the camera 44 to theother terminals 3 via the conference server 1 (step S13). The frame rateof the image data is 30 fps, for example. The CPU 31 encodes the imagedata every 1/30^(th) of a second, for example, in line with the framerate, and transmits the encoded image data to the other terminals 3.When it is not the timing to transmit the image data (no at step S13),the CPU 31 advances the processing to step S18. When the processing toterminate the remote conference has not been performed in the mainprocessing (no at step S18), the CPU 31 returns the processing to stepS11 and repeatedly performs the processing from step S11 to step S18.

During the time that the video transmission/reception processing isbeing repeatedly performed, when it becomes the timing to transmit theimage data (yes at step S13), the CPU 31 determines whether thecomposite image data is stored in the RAM 33 (step S15). The compositeimage data is generated in the camera control processing to be explainedlater (refer to FIG. 4), by combining a plurality of extracted imagedata that are extracted from the image data into single image data inaccordance with conditions. The composite image data is stored in apredetermined storage area of the RAM 33. When the composite image datais not stored in the RAM 33 (no at step S15), the CPU 31 performsprocessing to transmit the image data captured by the camera 44 to theother terminals 3. In this case, of image data stored in a camera buffer(to be explained later), the CPU 31 reads out the image data of a framethat is to be transmitted. The CPU 31 inputs the image data to the imageencoder and acquires the encoded image data that is encoded inaccordance with the predetermined video parameters. The CPU 31 storesthe encoded image data in the transmission buffer. As described above,the CPU 31 transmits the encoded image data stored in the transmissionbuffer to the conference server 1 in accordance with the OS (step S16).The CPU 31 advances the processing to step S18 and continuously performsthe video transmission/reception processing during the period in whichthe terminal 3 is participating in the remote conference.

At the timing to transmit the image data, when the composite image datais stored in the RAM 33 (yes at step S15), the CPU 31 performsprocessing to transmit the composite image data to the other terminals3. The CPU 31 inputs the composite image data read out from the RAM 33into the image encoder and acquires the encoded image data that isencoded in accordance with the predetermined video parameters. The CPU31 stores the encoded image data in the transmission buffer. Asdescribed above, the CPU 31 transmits the encoded image data of thecomposite image data stored in the transmission buffer to the conferenceserver 1 in accordance with the OS (step S17). The CPU 31 advances theprocessing to step S18 and continuously performs the videotransmission/reception processing during the period in which theterminal 3 is participating in the remote conference.

When the conference termination processing is performed in the mainprocessing and the conference termination notification event isgenerated while the video transmission/reception processing is beingrepeatedly performed (yes at step S18), the CPU 31 terminates theexecution of the video transmission/reception processing.

Next, the camera control processing that is started by the CPU 31 atstep S3 of the main processing of the remote conference program will beexplained. As shown in FIG. 4, in the camera control processing, the CPU31 transmits the command to the camera driver to activate the camera 44.The camera driver outputs an activation instruction to the camera 44,and activates the camera 44. The activation instruction includes theexposure setting instruction, and the camera 44 sets the exposure inaccordance with the exposure parameters and starts image capture (stepS21). The camera driver secures a storage area of the camera buffer inthe RAM 33. The camera buffer is a storage area to temporarily store theimage data captured by the camera 44. The camera driver startsprocessing that stores the image data acquired from the camera 44 viathe USB I/F 39 in the camera buffer.

Of the image data stored in the camera buffer, the CPU 31 reads out theimage data of a frame that is to be processed. The CPU 31 analyzes theimage data using a known algorithm, and performs face detectionprocessing (step S22). The face detection processing is, for example,processing in which, while scanning the image data using a search windowof a predetermined pixel size, feature quantities of pixels in thesearch window are compared using a discriminator that has been caused tolearn the feature quantities of the human face in advance, and it isthereby determined whether a human face is included in the searchwindow. Details of the face detection processing are known and anexplanation thereof is omitted here. When a face is detected from theimage data, the CPU 31 records a position of the detected face.

The CPU 31 determines whether two or more human faces have been detectedfrom the image data (step S23). When a face has not been detected fromthe image data, or when there is one human face detected (no at stepS23), the CPU 31 returns the processing to step S22, and performs theface detection processing on the image data of the next frame. When twoor more human faces have been detected from the image data (yes at stepS23), the CPU 31 extracts, from the image data, extracted image datathat is cut as an image of a predetermined range that includes positionsof the detected faces, and stores the extracted image data in the RAM 33(step S25).

For example, as shown in FIG. 8, an example is given of a case in whichfour people 51 to 54 are included in an image 50 represented by theimage data. The people 51 to 54 are respectively facing the frontsurface side of the image 50 (the front side of the drawing) and arearranged one above the other in two rows on the right and the left. Thepeople 51 to 54 are illuminated by light coming through a window 55 thatis shown in an upper right corner portion of the image 50. Faces 56 and57 of the people 51 and 52 who are side-by-side in the left row in theimage 50 are illuminated by the light coming through the window 55 andthe display of the faces 56 and 57 is bright. The face 56 of the person51 who is positioned in a location closer to the window 55 than theperson 52 is displayed more brightly than the face 57 of the person 52.For faces 58 and 59 of the people 53 and 54 who are side-by-side in theright row in the image 50, the backs of their heads are illuminated bythe light coming through the window 55, and the front side is in shadow.The display of the faces 58 and 59 are therefore darker than the displayof the faces 56 and 57 of the people 51 and 52. The face 59 of theperson 54 who is positioned in a location farther from the window 55than the person 53 is displayed more darkly than the face 58 of theperson 53.

In the face detection processing, the CPU 31 detects the four faces 56to 59 from the image 50 of the image data. The CPU 31 sets extractedareas 61 to 64 on the image 50, which are areas respectively includingthe faces 56 to 59. For example, the extracted area 61 is an area thatis encompassed by a rectangular frame having an area roughly 10 timesthe size of the area of the face 56, with the same aspect ratio as theimage 50. Within the image 50, the CPU 31 sets the extracted area 61 ina position in which the face 56 can be arranged roughly in the center ofthe area in the left-right direction and in an upper portion in theup-down direction. The CPU 31 sets the extracted areas 62 to 64 in asimilar manner.

As shown in FIG. 9, the CPU 31 stores extracted image data 66 to 69 inthe RAM 33, the extracted image data 66 to 69 respectively correspondingto the cut out portions of the extracted areas 61 to 64 from the image50. Images shown by each of the extracted image data 66 to 69respectively include the faces 56 to 59 of the people 51 to 54. Thefaces 56 to 59 are respectively influenced by the light coming throughthe window 55, and are displayed as faces having different brightness(luminance), including a light face, a dark face and a face ofbrightness in between light and dark. It should be noted that thesettings of the above-described extracted areas 61 to 64 are an example.For example, the CPU 31 may set the image 50 such that the size of theextracted areas 61 to 64 is slightly larger than the size of the faces56 to 59, and the faces 56 to 59 are arranged in the center of theextracted areas 61 to 64.

As shown in FIG. 4, the CPU 31 analyzes each of the extracted image datastored in the RAM 33 and determines a luminance value in the vicinity ofthe face (hereinafter referred to as a “face proximity luminance value”)of the extracted image data (step S26). The face proximity luminancevalue is a luminance value that represents a luminance value of theextracted image data. Note that luminance values of respective pixelsthat configure the extracted image data are referred to collectively asthe extracted image data luminance value. Further, in the presentembodiment, of the pixels that configure the extracted image data, anaverage value of the luminance values of pixels of a portion showing theface is determined to be the face proximity luminance value thatrepresents the luminance value of the extracted image data. Based on theposition of the face recorded in the face detection processing, the CPU31 acquires the luminance values of the pixels of the portion showingthe face, from the extracted image data. The CPU 31 calculates theaverage value of the acquired luminance values and determines the faceproximity luminance value, associates the average value with theextracted image data and stores the associated data in the RAM 33. TheCPU 31 respectively determines the face proximity luminance value forall of the extracted image data stored in the processing at step S25. Itshould be noted that the average value used for the face proximityluminance value need not necessarily be the arithmetical mean of theluminance values of the pixels of the portion showing the face. Forexample, the face proximity luminance value may be the luminance valueof a pixel positioned in the center of a luminance histogram (namely, amedian value), among the pixels of the portion showing the face. Forexample, the face proximity luminance value may be a highest luminancevalue (namely, the mode value), among the luminance values of the pixelsof the portion showing the face. In other words, the average value meansthe inclusion of the arithmetical mean, the median value or the modevalue. Further, for example, the face proximity luminance value may bethe average value of the luminance values of the pixels of the portionshowing the face and of the pixels surrounding that portion. Forexample, the face proximity luminance value may be the average value ofthe luminance values of all the pixels configuring the extracted imagedata. In addition, for example, the face proximity luminance value neednot necessarily be the calculated average value, and the luminance valueof a pixel whose coordinate position is central among the pixels of theportion showing the face may be used as the face proximity luminancevalue.

The CPU 31 performs a sub-routine of reference luminance valuedetermination processing (step S27). The reference luminance value is aluminance value that is used as a reference when adjusting the luminancevalue of each of the extracted image data based on the face proximityluminance value. In discrete image luminance adjustment processing(refer to FIG. 6) that will be explained later, the CPU 31 performscorrection such that the face proximity luminance value of each of theextracted image data falls within an allowable luminance range that hasthe reference luminance value as its central value. The allowableluminance range is a range of luminance values that ranges between aluminance upper limit value and a luminance lower limit value with thereference luminance value as the central value. The luminance upperlimit value is a value that is obtained by adding a predetermined secondallowable value to the reference luminance value. The luminance lowerlimit value is a value that is obtained by subtracting the secondallowable value from the reference luminance value. In the referenceluminance value determination processing, when an input has beenreceived of an operation to specify the image data displayed on themonitor 41, the CPU 31 can determine the reference luminance value thatis used as the reference when correcting the luminance value of theextracted image data. The determination of the reference luminance valueis performed based on the face proximity luminance value of the imagedata that has been specified (hereinafter also referred to as “specifiedimage data”) or on the extracted image data.

As shown in FIG. 5, in the reference luminance value determinationprocessing, the CPU 31 determines whether the input to specify one ofthe images has been received, based on the operation on the inputportion 42 (step S41). Via the input portion 42, the CPU 31 receives theinput of the operation that specifies one of the image data from amongthe image data of the other terminals 3 and the image data of the oneterminal 3 which are displayed on the monitor 41. Further, when theimage data displayed on the monitor 41 is the composite image data, theCPU 31 receives, via the input portion 42, the input of the operationspecifying one of the extracted image data from among the plurality ofextracted image data that configure the composite image data. When theinput of the operation specifying the one image data or the oneextracted image data has been received (yes at step S41), the CPU 31acquires the face proximity luminance value of the specified image data(step S42). The face proximity luminance value of the specified imagedata is acquired by processing that is similar to that of theabove-described case in which the face proximity luminance value of theextracted image data is determined. In other words, the CPU 31 performsthe face detection processing on the specified image data. The CPU 31acquires the luminance values of the pixels of the portion showing thedetected face from the specified image data and calculates the averagevalue of the acquired luminance values. The CPU 31 sets the calculatedaverage value as the reference luminance value and stores the referenceluminance value in the RAM 33 (step S43). Note that, when a face is notdetected in the specified image data, the CPU 31 may set the averagevalue of the luminance values of all the pixels of the image data as thereference luminance value. Further, when a plurality of faces aredetected in the specified image data, the CPU 31 may set an averagevalue of each of the face proximity luminance values of the plurality offaces as the reference luminance value. After setting the referenceluminance value, the CPU 31 returns the processing to the camera controlprocessing (refer to FIG. 4) of the remote conference program, andadvances the processing to step S28.

In addition, also when the input of the operation specifying the imagedata or the extracted image data has been received on one of the otherterminals 3, the CPU 31 of the one terminal 3 can determine thereference luminance value that is used as the reference when correctingthe luminance value of the extracted image data. The determination ofthe reference luminance value is performed based on the face proximityluminance value of the specified image data. When the input of theoperation to specify the image data or the extracted image data has notbeen received by the input portion 42 of the one terminal 3 (no at stepS41), the CPU 31 determines whether a notification specifying the imagedata or the extracted image data (hereinafter also referred to as“specification information”) has been received from one of the otherterminals 3 via the conference server 1 (step S45). Note that theterminal 3 that participates in the conference using the user ID towhich special authorization (such as an authorization to chair theconference etc.) has been assigned, for example, can transmit thespecification information to the other terminals 3 via the conferenceserver 1. When the CPU 31 of the one terminal 3 has received thespecification information (yes at step S45), the CPU 31 acquires theface proximity luminance value of the specified image data in a similarmanner to that at step S42 (step S46). The CPU 31 sets the faceproximity luminance value of the specified image data as the referenceluminance value and stores the reference luminance value in the RAM 33(step S47). After setting the reference luminance value, the CPU 31returns the processing to the camera control processing (refer to FIG.4) of the remote conference program, and advances the processing to stepS28.

When the input of the operation specifying the image data or theextracted image data has not been received (no at step S41) and when thespecification information has also not been received (no at step S45),the CPU 31 determines whether the reference luminance value has been set(step S48). When the reference luminance value determination processingis performed for the first time, the reference luminance value has notbeen set (no at step S48). Thus, the CPU 31 sets a predeterminedstandard luminance value as the reference luminance value and stores thereference luminance value in the RAM 33 (step S50). The CPU 31 returnsthe processing to the camera control processing (refer to FIG. 4) of theremote conference program, and advances the processing to step S28. Whenthe reference luminance value determination processing is performed fromthe second time onward, the reference luminance value has been set (yesat step S48). Thus, the CPU 31 returns the processing to the cameracontrol processing (refer to FIG. 4) of the remote conference program,and advances the processing to step S28.

As shown in FIG. 4, when the CPU 31 returns the processing from thereference luminance value determination processing to the camera controlprocessing, it performs the sub-routine of the discrete image luminanceadjustment processing (step S28). The discrete image luminanceadjustment processing is processing that corrects the face proximityluminance value of each of the extracted image data to be within theallowable luminance range that has the reference luminance value as itscentral value. As shown in FIG. 6, in the discrete image luminanceadjustment processing, the CPU 31 selects one of the plurality ofextracted image data stored in the RAM 33 by the processing at step S25,and takes the selected extracted image data as a target for luminancevalue adjustment (step S51). The CPU 31 reads out the face proximityluminance value of the selected extracted image data from the RAM 33,and compares the face proximity luminance value with the luminance upperlimit value of the allowable luminance range (step S52). As describedabove, the allowable luminance range has the reference luminance valuedetermined in the reference luminance value determination processing(refer to FIG. 5) as its central value, and is a range of luminancevalues between the luminance upper limit value and the luminance lowerlimit value. The exposure upper limit value is a value that is obtainedby adding the predetermined second allowable value to the referenceluminance value. The exposure lower limit value is a value that isobtained by subtracting the second allowable value from the referenceluminance value.

When the face proximity luminance value is equal to or greater than theluminance upper limit value of the allowable luminance range (yes atstep S52), the CPU 31 subtracts the luminance upper limit value of theallowable luminance range from the face proximity luminance value andthus calculates a difference value (step S53). In this case, thedifference value is a positive value. The CPU 31 stores the differencevalue in the RAM 33 (step S57) and corrects the luminance value of theextracted image data based on the difference value (step S58).Specifically, the CPU 31 performs processing that changes each of theluminance values, for all the pixels configuring the extracted imagedata, to the luminance value calculated by subtracting the differencevalue from the original luminance value. The luminance value of all thepixels is corrected to a smaller value by an absolute value of thedifference value, and thus the face proximity luminance value of theextracted image data after the correction becomes the luminance upperlimit value of the allowable luminance range, and falls within theallowable luminance range. When there is not yet selected extractedimage data in the RAM 33 and the correction with respect to each of theextracted image data corresponding to the number of faces that have beendetected is not complete (no at step S60), the CPU 31 returns theprocessing to step S51.

When the face proximity luminance value of the extracted image dataselected by the CPU 31 is less than the luminance upper limit value ofthe allowable luminance range (no at step S52) and is equal to or lessthan the luminance lower limit value of the allowable luminance range(yes at step S55), the CPU 31 calculates a difference value that isobtained by subtracting the luminance lower limit value of the allowableluminance range from the face proximity luminance value (step S56). Inthis case, the difference value is a negative value. The CPU 31 storesthe difference value in the RAM 33 (step S57) and corrects the luminancevalue of the extracted image data based on the difference value (stepS58). In a similar manner to that described above, the CPU 31 performsprocessing that changes each of the luminance values, for all the pixelsconfiguring the extracted image data, to the luminance value calculatedby subtracting the difference value from the original luminance value.As the difference value is a negative value, the luminance value of allthe pixels is corrected to a greater value by an absolute value of thedifference value. The face proximity luminance value of the extractedimage data after the correction becomes the luminance lower limit valueof the allowable luminance range, and falls within the allowableluminance range. The CPU 31 stores the extracted image data for whichthe luminance value has been corrected in the RAM 33. In a similarmanner to that described above, when the correction is not complete forall of the extracted image data (no at step S60), the CPU 31 returns theprocessing to step S51.

When the face proximity luminance value of the extracted image dataselected by the CPU 31 is less than the luminance upper limit value ofthe allowable luminance range (no at step S52) and is greater than theluminance lower limit value of the allowable luminance range (no at stepS55), the face proximity luminance value is a value that is within theallowable luminance range. In this case, the CPU 31 does not perform thecorrection on the luminance value of the extracted image data andadvances the processing to step S60. When the CPU 31 completes thecorrection for all the extracted image data (yes at step S60), the CPU31 returns the processing to the camera control processing (refer toFIG. 4) of the remote conference program and advances the processing tostep S30.

As shown in FIG. 4, after the CPU 31 returns the processing to thecamera control processing from the discrete image luminance adjustmentprocessing, the CPU 31 performs a sub-routine of image capture exposureadjustment processing (step S30). The image capture exposure adjustmentprocessing is processing that outputs to the camera driver the changecommand to change the exposure of the camera 44, in order to minimize adifference between the face proximity luminance value of the extractedimage data and the reference luminance value. As shown in FIG. 7, in theimage capture exposure adjustment processing, the CPU 31 calculates anaverage value of the plurality of difference values stored in the RAM 33by the processing at step S57 (step S71). The magnitude of averagedisplacement, from the reference luminance value, of the face proximityluminance value of the extracted image data for which the face proximityluminance value did not fall within the allowable luminance range, iscalculated as the average value of the difference values. The CPU 31calculates an adjustment value (step S72). The adjustment value is avalue obtained by displacing the reference luminance value by theaverage value for example, and is used to determine whether to performadjustment of the exposure correction value based on the average value.The CPU 31 adds the average value to the reference luminance value andthus calculates the adjustment value.

In a similar manner to that described above, when there is no input ofthe operation specifying the image data or the extracted image data, andthe specification information has not been received (no at step S73),the CPU 31 adjusts the exposure correction value in accordance with theadjustment value (step S80). The average value that is the basis ofcalculation of the adjustment value is reflected in the exposurecorrection value. As described above, the exposure correction value is avalue of the change amount of the luminance value of the image datacaptured by the camera 44 and is specified by the CPU 31. Specifically,the exposure correction value is a value that specifies the changeamount of the luminance value based on the average value before thechange in exposure, such that the average value of the plurality ofdifference values becomes a value less than the average value before thechange in exposure (and is preferably zero). The plurality of differencevalues are the respective difference values between the referenceluminance value and the face proximity luminance values of the pluralityof extracted image data extracted from the image data of the camera 44after the change in exposure. The control to correct the exposure of thecamera 44 is performed by the camera driver based on the exposurecorrection value. The CPU 31 transmits the change command including theexposure correction value to the camera driver (step S81). As describedabove, when the camera driver receives the change command, the cameradriver changes the exposure parameters in accordance with the exposurecorrection value and outputs the exposure setting instruction to thecamera 44. The camera 44 corrects the exposure in accordance with theexposure setting instruction, and outputs image data captured using theexposure after the correction. The CPU 31 returns the processing to thecamera control processing (refer to FIG. 4) of the remote conferenceprogram, and advances the processing to step S31.

In the processing at step S73, when there is the input of the operationspecifying the image data or the extracted image data, or when thespecification information is received (yes at step S73), the CPU 31compares the adjustment value with an exposure correction range that isbased on the face proximity luminance value of the specified image data(step S75 and step S76). The exposure correction range is a range ofluminance values that has the face proximity luminance value of thespecified image data as its central value and that ranges between anexposure upper limit value and an exposure lower limit value. Theexposure upper limit value is a value that is obtained by adding apredetermined first allowable value to the face proximity luminancevalue. The exposure lower limit value is a value that is obtained bysubtracting the predetermined first allowable value from the faceproximity luminance value. When the adjustment value is greater than theexposure upper limit value of the exposure correction range (yes at stepS75), the CPU 31 adjusts the exposure correction value in accordancewith the exposure upper limit value (step S77). Namely, when the faceproximity luminance value of the image data after the change in exposureis a large value that exceeds the face proximity luminance value of thespecified image data by an amount in excess of the first allowablevalue, the CPU 31 adjusts the exposure correction value in accordancewith the exposure upper limit value. The adjustment of the exposurecorrection value in accordance with the exposure upper limit value is anadjustment that is made such that the face proximity luminance value ofthe image data becomes a value that is obtained by adding the firstallowable value to the face proximity luminance value of the specifiedimage data. The CPU 31 transmits the change command including theexposure correction value to the camera driver (step S81). The CPU 31returns the processing to the camera control processing (refer to FIG.4) of the remote conference program, and advances the processing to stepS31.

Further, when the adjustment value is less than the exposure lower limitvalue of the exposure correction range (no at step S75 and yes at stepS76), the CPU 31 adjusts the exposure correction value in accordancewith the exposure lower limit value (step S78). Namely, when the faceproximity luminance value of the image data after the change in exposureis a value that is less than the face proximity luminance value of thespecified image data by an amount in excess of the first allowablevalue, the CPU 31 adjusts the exposure correction value in accordancewith the exposure lower limit value. The adjustment of the exposurecorrection value in accordance with the exposure lower limit value is anadjustment that is made such that the face proximity luminance value ofthe image data becomes a value that is obtained by subtracting the firstallowable value from the face proximity luminance value of the specifiedimage data. The CPU 31 transmits the change command including theexposure correction value to the camera driver (step S81). The CPU 31returns the processing to the camera control processing (refer to FIG.4) of the remote conference program, and advances the processing to stepS31.

When the adjustment value is a value within the exposure correctionrange (no at step S75 and no at step S76), the CPU 31 advances theprocessing to step S80. In a similar manner to that described above,when the CPU 31 adjusts the exposure correction value in accordance withthe adjustment value (step S80) and transmits the change commandincluding the exposure correction value to the camera driver (step S81),the CPU 31 then advances the processing to step S31 of the cameracontrol processing (refer to FIG. 4).

As shown in FIG. 4, when the CPU 31 returns the processing to the cameracontrol processing from the image capture exposure adjustmentprocessing, the CPU 31 combines the extracted image data (step S31). TheCPU 31 reads out the plurality of extracted image data for which theluminance value has been corrected in the processing at step S58 fromthe RAM 33, and generates the composite image data that represents acomposite image in which images represented by each of the extractedimage data are arranged side by side in a single image formation area.The CPU 31 stores the composite image data in the RAM 33.

As shown in FIG. 10, the CPU 31 generates the composite image data thatrepresents a single composite image 70 in which images represented byeach of the extracted image data 66 to 69 are arranged in a tileformation, for example. The faces 56 to 59 of the people 51 to 54respectively displayed in the images represented by each of theextracted image data 66 to 69 are faces having a brightness that hasbeen corrected so as to be closer to the reference luminance value thanthe respective luminance values before the respective luminance valuesare corrected. Specifically, the differences between the respective faceproximity luminance values of the faces 56 to 59 are less than beforethe correction of the luminance values. Thus, in the composite image 70represented by the composite image data, the variations in the luminancevalues of each of the extracted image data 66 to 69 are less than beforethe correction of the luminance values. As a result, in the compositeimage 70, the variations in the luminance values of the image as a wholeare less than before the correction of the luminance values, and incomparison to a case in which the combining of the images is performedwithout performing the luminance value correction, the composite image70 is an image in which the unnaturalness of changes in the luminancevalues within the image is alleviated.

As shown in FIG. 4, when the processing to terminate the remoteconference has not been performed in the main processing (no at stepS32), the CPU 31 returns the processing to step S22. The CPU 31 extractsthe extracted image data from the not yet processed image data, inaccordance with the conditions, and repeatedly performs the processingfrom step S22 to step S32 to generate the composite image data. When theremote conference terminate processing is performed in the mainprocessing and the notification event for the termination of the remoteconference has occurred while the processing from step S22 to step S32is being repeatedly performed (yes at step S32), the CPU 31 transmits acommand to the camera driver to stop the driving of the camera 44 (stepS33). The camera driver outputs a stop instruction to the camera 44 andstops the driving of the camera 44. The CPU 31 terminates the executionof the camera control processing.

As explained above, the CPU 31 can reduce the differences in theluminance values of the plurality of extracted image data that configurethe composite image data by correcting the luminance values of therespectively corresponding extracted image data with the plurality ofdifference values based on the plurality of face proximity luminancevalues and the reference luminance value. As a result, the CPU 31 canprovide the composite image in which variations in the luminance betweenthe extracted image data are small, even if the changes in the luminancewithin the image are not based on the direction of light from the lightsource, thus providing the composite image that can suppressunnaturalness of the image as a whole. Further, by determining theexposure correction value based on the plurality of difference values,the CPU 31 can change the exposure of the camera 44 in accordance withthe change command that is based on the exposure correction value.Specifically, the exposure correction value is determined such that theaverage value of the plurality of difference values can be made a valuethat is less than the average value before the change in exposure (andthat is preferably zero). In other words, the exposure correction valueis determined such that the average value of the plurality of faceproximity luminance values after the change in exposure can be made avalue that is closer to the reference luminance value than before thechange in exposure (and that is preferably equal to the referenceluminance value). In this way, the face proximity luminance values thatare respectively determined for the plurality of extracted image datathat are determined with respect to the image data of the camera 44after the change in exposure each come closer to the reference luminancevalue. Thus, even without correcting the respective luminance values ofthe plurality of extracted image data, in the composite image, the CPU31 can minimize the differences in the luminance values of therespective extracted image data. As a result, the CPU 31 can reduce aload of the arithmetic calculation to determine the exposure correctionvalue.

The CPU 31 can calculate the adjustment value involved in thedetermination of the exposure correction value based on the averagevalue of the plurality of difference values. Thus, among the images ofthe plurality of extracted image data determined with respect to theimage data acquired from the camera 44 after the change in exposure,there are no extremely bright images and extremely dark images. As aresult, even if the changes in the luminance within the image are notbased on the direction of the light from the light source, the CPU 31can provide the composite image in which the variations in the luminancebetween the extracted image data are small, thus providing the compositeimage that can suppress unnaturalness of the image as a whole.

The CPU 31 can correct the luminance value of each of the extractedimage data such that the respective face proximity luminance values ofthe plurality of extracted image data determined with respect to theimage data acquired from the camera 44 after the change in exposure comecloser to the face proximity luminance value of the specified imagedata. As a result, the CPU 31 can generate the composite image that isclose to the face proximity luminance value of the specified image data.When the specified image data is selected based on the input of theoperation on the input portion 42, the CPU 31 can cause the faceproximity luminance values of the plurality of extracted image datadetermined with respect to the image data acquired from the camera 44after the change in exposure to come closer to the face proximityluminance value of the specified image data. As a result, the CPU 31 cangenerate the composite image to easily distinguish the images of each ofthe extracted image data, in accordance with characteristics of adisplay screen of the terminal 3. Further, when the specified image datais selected based on the specification information received from one ofthe other terminals 3, the CPU 31 of the one terminal 3 can cause theface proximity luminance values of the plurality of extracted image datadetermined with respect to the image data acquired from the camera 44after the change in exposure to come closer to the face proximityluminance value of the specified image data selected by the otherterminal 3. As a result, the CPU 31 can generate the composite image toeasily distinguish the images of each of the extracted image data, inaccordance with characteristics of a display screen of the otherterminal 3.

When the specified image data has been selected, the CPU 31 candetermine the exposure correction value in accordance with the faceproximity luminance value of the specified image data. Further, when theadjustment value is outside the allowable luminance range of thereference luminance value, the CPU 31 can determine the exposurecorrection value based on the exposure upper limit value or the exposurelower limit value. As a result, for the images of the plurality ofextracted image data determined with respect to the image data acquiredfrom the camera 44 after the change in exposure, there are no extremelybright images and extremely dark images with respect to the specifiedimage data. Therefore, even if the changes in the luminance within theimage are not based on the direction of the light from the light source,the CPU 31 can provide the composite image in which the variations inthe luminance between the extracted image data are small, thus providingthe composite image that can suppress unnaturalness of the image as awhole.

When the absolute value of the difference value is equal to or greaterthan the second allowable value, the CPU 31 performs arithmeticcalculation to determine the exposure correction value. Specifically,when the absolute value of the difference value is less than the secondallowable value (in other words, it is within the allowable luminancerange, namely, the face proximity luminance value is less than theluminance upper limit value and greater than the luminance lower limitvalue), the CPU 31 need not determine the difference value with respectto the extracted image data, and further need not perform the arithmeticcalculation to determine the exposure correction value. As a result, itis possible to reduce the load involved in the arithmetic calculation.

When the face proximity luminance value is equal to or greater than theluminance upper limit value, the CPU 31 performs the correction based onthe luminance upper limit value, and similarly, when the face proximityluminance value is equal to or lower than the reference luminance value,the CPU 31 performs the correction based on the luminance lower limitvalue. As a result, the CPU 31 can correct the luminance value of theextracted image data with a minimum magnitude of change such that theface proximity luminance value is included within the allowableluminance range, without a large change with respect to the luminancevalue before the correction. Therefore, the CPU 31 can acquire, as theimages of each of the extracted image data in the composite image data,the images that appropriately reflect the respective luminances of theimages of each of the corresponding extracted image data in the originalimage data.

The CPU 31 can determine, as the extracted image data, the predeterminedarea that includes the position of the face. Thus, in the compositeimage that is formed by combining the images of the plurality ofextracted image data determined with respect to the image data acquiredfrom the camera 44 after the change in exposure, there are no images inwhich the face image is extremely bright and no images in which the faceimage is extremely dark. As a result, the CPU 31 can generate thecomposite image data in which the faces can be easily distinguished.

Note that the present disclosure is not limited to the above-describedembodiment and various changes are possible. For example, the terminal 3is not limited to a laptop or tower type PC, and may be a notebook PC, asmart phone, a tablet terminal or the like. Specifically, it issufficient that the terminal 3 be a terminal device that can execute theremote conference program, can be connected to the network 8, canperform the image capture using the camera 44, and can display the imageon the monitor 41 using the display control portion 34. Further, atleast one of the microphone 43, the speaker 45, the camera 44 and themonitor 41 may be built into the terminal 3.

The CPU 31 is not limited to changing the brightness of the image databy changing the luminance value only, and the CPU 31 may change thebrightness of the image data by changing other parameters, such aslightness, saturation, a gamma value etc. In addition, as the extractedimage data, the CPU 31 may cut out only an area of the human face and,after correcting the luminance value, may generate the composite imagedata by overwriting the extracted location of the original image data.

What is claimed is:
 1. A non-transitory computer-readable storage mediumstoring computer-readable instructions that, when executed by aprocessor of a terminal device, which is configured to performtransmission and reception of various data with a plurality of terminaldevices connected via a network, causes the terminal device to performprocesses comprising: a first generating processing generating aplurality of extracted image data from captured image data acquired by acamera connected to the terminal device, the plurality of extractedimage data corresponding to portions in the captured image data, each ofthe extracted image data including a respective target to be extractedfrom the captured image data; a first determination processingdetermining a plurality of representative luminance values respectivelyrepresenting luminances of the plurality of extracted image data; asecond determination processing determining a plurality of luminancecorrection values corresponding to the plurality of extracted imagedata, respectively, based on a reference luminance value and theplurality of representative luminance values determined in the firstdetermination processing, the reference luminance value being areference value of luminance; a storage processing storing the pluralityof luminance correction values determined in the second determinationprocessing in a storage device provided in the terminal device; a secondgenerating processing generating composite image data from the pluralityof extracted image data, the respective luminance of each of theplurality of extracted image data having been corrected using acorresponding luminance correction value of the plurality of luminancecorrection values stored in the storage device in the storageprocessing; and a third determination processing determining, based onthe plurality of luminance correction values stored in the storagedevice in the storage processing, an exposure correction value tocorrect an exposure of the camera, the third determination processingcomprising: first calculation processing calculating an average value ofthe plurality of luminance correction values, and second calculationprocessing calculating a first addition value by adding the averagevalue calculated in the first calculation processing and the referenceluminance value, wherein the exposure correction value is determinedbased on the first addition value calculated in the second calculationprocessing.
 2. The non-transitory computer-readable storage mediumaccording to claim 1, wherein the instructions, when executed, furthercause the terminal device to perform: first transmission processingtransmitting, to the camera, a change command changing the exposure ofthe camera, based on the exposure correction value determined in thethird determination processing.
 3. The non-transitory computer-readablestorage medium according to claim 1, wherein the instructions, whenexecuted, further cause the terminal device to perform: selectionprocessing selecting at least one of the extracted image data as aselected image data from among the extracted image data generated by thefirst generating processing; and setting processing setting, as thereference luminance value, a respective one of the representativeluminance values representing the luminance of the selected image data.4. The non-transitory computer-readable storage medium according toclaim 3, wherein the selection processing comprises selecting at leastone of the plurality of extracted image data as the selected image databased on an input of an operation received by an input portion providedin the terminal device.
 5. The non-transitory computer-readable storagemedium according to claim 3, wherein the instructions, when executed,further cause the terminal device to perform: second transmissionprocessing transmitting the composite image data to another terminaldevice via the network, and wherein the selection processing comprisesselecting at least one of the plurality of extracted image data as theselected image data based on specification information received from theother terminal device, the specification information specifying theselected image data selected from among the plurality of extracted imagedata included in the composite image data on the other terminal device.6. The non-transitory computer-readable storage medium according toclaim 3, wherein: the third determination processing further comprises:first judgment processing judging whether the selected image data isselected, and second judgment processing judging whether the firstaddition value is included in a pre-set specific range, the referenceluminance value being based on the luminance of the selected image dataand being set in the setting processing, and wherein the exposurecorrection value is determined based on a second addition value when thefirst judgment processing judging that the selected image data isselected and the second judgment processing judging that the firstaddition value is not included within the specific range, the secondaddition value being a value obtained by adding the reference luminancevalue and a predetermined first allowable value.
 7. The non-transitorycomputer-readable storage medium according to claim 1, wherein: thesecond determination processing further comprises third judgmentprocessing judging whether an absolute value of a difference valuebetween one of the representative luminance values and the referenceluminance value is equal to or greater than a second allowable value,and the luminance correction value is determined with respect to acorresponding one of the extracted image data when the third judgmentprocessing judges that the absolute value of the difference value isequal to or greater than the second allowable value.
 8. Thenon-transitory computer-readable storage medium according to claim 7,wherein the second determination processing comprises, for each of theluminance correction values: determining the respective luminancecorrection value based on a luminance upper limit value when acorresponding one of the representative luminance values is equal to orgreater than the luminance upper limit value, the luminance upper limitvalue being obtained by adding the second allowable value to thereference luminance value, and determining the respective luminancecorrection value based on a luminance lower limit value when acorresponding one of the representative luminance values is equal to orless than the luminance lower limit value, the luminance lower limitvalue being obtained by subtracting the second allowable value from thereference luminance value.
 9. The non-transitory computer-readablestorage medium according to claim 1, wherein the instructions furtherperform a process comprising: detection processing detecting a positionof a face in the captured image data, and wherein the first generatingprocessing includes generating the plurality of extracted image data toinclude the detected position of the face.
 10. A terminal deviceperforming transmission and reception of various data with a pluralityof terminal devices configured to be connected via a network, theterminal device comprising: a processor, and a memory storingcomputer-readable instructions, the instructions, when executed by theprocessor, perform processes comprising: a first generating processinggenerating a plurality of extracted image data from captured image dataacquired by a camera connected to the terminal device, the plurality ofextracted image data corresponding to portions in the captured imagedata, and each of the extracted image data including a respective targetto be extracted from the captured image data; a first determinationprocessing determining a plurality of representative luminance valuesrespectively representing luminances of the plurality of extracted imagedata; a second determination processing determining a plurality ofluminance correction values corresponding to the plurality of extractedimage data, respectively, based on a reference luminance value and theplurality of representative luminance values determined in the firstdetermination processing, the reference luminance value being areference value of luminance; a storage processing storing the pluralityof luminance correction values determined in the second determinationprocessing in the memory; a second generating processing generatingcomposite image data from the plurality of extracted image data, therespective luminance of each of the plurality of extracted image datahaving been corrected using a corresponding luminance correction valueof the plurality of luminance correction values stored in the memory;and a third determination processing determining, based on the pluralityof luminance correction values stored in the memory in the storageprocessing, an exposure correction value to correct an exposure of thecamera, the third determination processing comprising: first calculationprocessing calculating an average value of the plurality of luminancecorrection values, and second calculation processing calculating a firstaddition value by adding the average value calculated in the firstcalculation processing and the reference luminance value, wherein theexposure correction value is determined based on the first additionvalue calculated in the second calculation processing.
 11. The terminaldevice according to claim 10, wherein the instructions further performprocesses comprising: transmission processing transmitting, to thecamera, a change command changing the exposure of the camera, based onthe exposure correction value determined in the third determinationprocessing.
 12. A method executed by a processor of a terminal device,which performs transmission and reception of various data with aplurality of terminal devices configured to be connected via a network,the method comprising: first generating a plurality of extracted imagedata from captured image data acquired by a camera connected to theterminal device, the plurality of extracted image data corresponding toportions in the captured image data, each of the plurality of extractedimage data including a respective target to be extracted from thecaptured image data; first determining a plurality of representativeluminance values respectively representing luminances of the pluralityof extracted image data; second determining a plurality of luminancecorrection values corresponding to the plurality of extracted imagedata, respectively, based on a reference luminance value and theplurality of representative luminance values determined in the firstdetermining, the reference luminance value being a reference value ofluminance; storing the plurality of luminance correction valuesdetermined in the second determining in a storage device provided in theterminal device; second generating composite image data from theplurality of extracted image data, the luminance of each of theplurality of extracted image data having been corrected using acorresponding luminance correction value of the plurality of luminancecorrection values stored in the storage device; and third determining,based on the plurality of luminance correction values stored in thestorage processing, an exposure correction value to correct an exposureof the camera, the third determining comprising: first calculationprocessing calculating an average value of the plurality of luminancecorrection values, and second calculation processing calculating a firstaddition value by adding the average value calculated in the firstcalculation processing and the reference luminance value, wherein theexposure correction value is determined based on the first additionvalue calculated in the second calculation processing.
 13. The methodaccording to claim 12, further comprising: transmitting, to the camera,a change command changing the exposure of the camera, based on theexposure correction value determined in the third determining.